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Simultaneous Segmentation of Multiple or Composed Objects by Mesh Adaptation 



The present invention relates to a method for the simultaneous segmentation of multiple 
or composed objects in an image. Furthermore, the present invention relates to an image 
processing device adapted for executing the method for the simultaneous segmentation 
of multiple or composed objects in an image and a computer program comprising 
instruction code for executing the method for the simultaneous segmentation of multiple 
or composed objects. 



10 Segmentation methods are used to derive geometric models of, for example, organs or 
bones from volumetric image data, such as CT, MR or US images. Such geometric 
models are required for a variety of medical applications, or generally in the field of 
pattern recognition. For medical or clinical applications, an important example is cardiac 
diagnosis, where geometric models of the_ventricles and the myocard of the heart are 

15 required for perfusion analysis, wall motion analysis and computation of the ejection 
fraction. Another important clinical application is radio-therapy planning, where the 
segmentation of multiple organs and bones in, for example, the prostrate region (femur 
heads, rectum, prostate, bladder) is necessary for the diagnosis and/or for the 
determination of treatment parameters. 

20 

Deformable models are a very general class of methods for the segmentation of 
structures in 3D images. Deformable models are known, e.g. from an article of T. 
Mclnerney et al. "Deformable models in medical image analysis: A survey", Medical 
Image Analysis, 1(2):91-108, 1996. 

25 

The basic principle of deformable models consist of the adaptation of flexible meshes, 
represented, for example, by triangles or simplexes, to the object of interest in an'image. 
For this, the model is initially placed near or on the object of interest in the image. This 
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may be done by a user. Then, coordinates of surface elements of the flexible mesh, such 
as the triangles, are iteratively changed until they he on or close to the surface of the 
object of interest. Such a method is described in further detail in J. Weese et al. "Shape 
constrained deformable models for 3D medical image segmentation" in 17 th International 
Conference on Information Processing in Medical Imaging QPMT), pages 380-387, 
Davies, CA, USA, 2001, Springer Verlag. 

Usually, several separate structures of interest, such as the femur head, the rectum, the 
prostate and the bladder in the prostate region, have to be segmented. Also, it often 
occurs that the structure of interest consists of several objects, such as two vertebra of 
the spine. In spite of having a certain spatial configuration, often, these objects are not 
connected. In such cases, adaptation may become a problem. 

If a plurality of objects are segmented one after the other, a correct segmentation of the 
15 individual objects is often not possible. Itoften occurs that the spatial relationship 
between the objects, for example, a minimum distance between two object boundaries 
cannot be maintained. 

It is also possible to carry out the segmentation process simultaneously for all objects. 
20 This can be done, for example, for reasons of speed and practicality. However, during a 
simultaneous segmentation of a plurality of objects, it often occurs that the spatial 
relationships are not maintained and are violated during mesh reconfiguration. This 
causes meshes which are intersecting or too close to each other, thus resulting in a 
wrong segmentation result. 



10 



25 



Accordingly, it is an object of the present invention to provide a method for the 
simultaneous segmentation of multiple or composed objects in an image which allows an 
improved simultaneous segmentation of multiple or composed objects. 



30 
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The above object is solved by a method for the simultaneous segmentation of multiple or 
composed objects in an image, wherein a deformable surface model is to be adapted to a 
first surface of a first object and a second surface of a second object and wherein the 
deformable surface model comprises a first partial deformable surface model and a 
5 second partial deformable surface model, comprising the steps of: 

(a) applying the first partial deformable surface model describing a structure of 
the first surface of the first object; 

(b) applying the second partial deformable surface model describing a structure 
of the second surface of the second object, wherein the first partial deformable surface 

10 model and the second partial deformable surface model have a prescribed spatial 
relationship corresponding to a spatial relationship of the first object and the second 
object; and 

(c) adapting the first partial deformable model to the first surface and the second 
partial deformable model to the second surface, wherein the prescribed spatial 

1 5 relationship of the first partial deformabIe_surface model and the second partial 
deformable surface model is used for the adaptation. 

In other words, in this method according to an exemplary embodiment according to the 
present invention, a priori knowledge of a spatial relationship of the first and second 
20 objects is used for adaptation. Advantageously, this method according to an exemplary 
embodiment of the present invention allows to carry out a simultaneous segmentation 
while maintaining spatial relationships between a plurality of objects during mesh 
configuration. Advantageously, this avoids intersecting meshes and/or meshes which are 
too close to each other. 

25 

According to an exemplary embodiment of the present invention as set forth in claim 2, 
the step of applying a priori knowledge of the spatial relationship of the first and second 
objects for adaptation in accordance with step (c) of claim 1, involves the use of an 
additional edge connecting the surfaces of the two deformable surface models, or a 
30 plurality of additional edges which are connecting vertices of the two deformable surface 
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models. The method according to claim 2 provides a simple methods which can be 
executed with a minimized computation effort. Furthermore, the use of additional edges 
can be integrated without further amendments into other segmentation methods, such as 
the one known from J. Weese et al. 

5 According to another exemplary embodiment of the present invention as set forth in 
claim 3, the additional edge between objects is a featureless triangle. This means that 
assuming a mesh with triangles, an additional triangle is denned between a vertex of the 
first deformable surface model and a vertex of the second deformable surface model. 

10 The additional triangle is introduced such that when numbering the vertices of the 
triangle with 1, 2, 3, the vertices 1 and 3 of the additional triangle are identical. Thus, 
the additional triangle does not have any surface. Accordingly, this additional triangle 
does not contribute to a calculation of an external energy during segmentation. The 
motivation for no external energy (U. not searching for a surface feature) is that a 

15 connection between two deformable surface model does not correspond to an object 
surface. 

Further exemplary embodiments according to the present invention, as set forth in claims 
4 and 5 provide, for a fast and efficient simultaneous segmentation of the two objects, 
20 while rninimizing computation efforts. 

According to another exemplary embodiment of the present invention as set forth in 
claim 6, an extended internal energy relating to vectorial difference of additional edges 
and approximately a distance between the two objects is introduced into the 
25 segmentation. A minimization of the internal energy with the extended internal energy 
allows to keep the spatial relationships of the two objects. 

Claim 7 relates to an exemplary embodiment of an image processing device suitably 
adapted for executing the method according to the present invention. Advantageously, 
30 this image processing device allows a very accurate simultaneous segmentation while 
avoiding intersecting meshes and while avoiding a violation of spatial relationships of the 
objects. 
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Claim 8 relates to an exemplary embodiment of a computer program executing the 
method according to the present invention. Advantageously, this computer program 
allows for a simultaneous segmentation of multiple or composed objects, without the 
occurrence of intersecting meshes or a violation of spatial relationships between the 
5 objects. 

It is the gist of the present invention, to include a priori knowledge concerning a spatial 
relationship of the two objects into the segmentation. For this, additional connections 
are introduced between vertices into a shape model which only contribute to the internal 
D energy, but are not used to detect feature points and therefore do not contribute to the 
external energies. If, for example, the distance between two surfaces is approximately 
known, additional connections between these surfaces are introduced. Maintaining the 
length of these connections (i.e. minimizing an extended internal energy) will minimize 
the deviation of the original distance between the objects. 

These and other aspects of the present invention will become apparent from and 
elucidated with reference to the embodiments described hereinafter. 

Exemplary embodiments of the present invention will be described in the following, with 
reference to the following drawings: 

Fig. 1 shows a schematic representation of an image processing device according to an 
exemplary embodiment of the present invention adapted to execute a method 
according to an exemplary embodiment of the present invention. 

Fig. 2 shows a flow chart of an exemplary embodiment of a method according to the 
present invention. 

Fig. 3 shows a triangular surface model of two vertebrae of the spine for explaining an 
exemplary embodiment of the present invention. 




PHDE020180 EP-P 



Fig. 4 shows two vertebrae with additional connections between upper and lower 
vertebra for explaining an exemplary embodiment of the present invention. 

Fig. 5 shows a simplified drawing for explaining a featureless vertex connection in 
5 accordance with an exemplary embodiment of the present invention. 

Figure 1 shows a simplified schematic representation of an exemplary embodiment of an 
image processing device in accordance with the present invention. In Fig. 1, there is 
shown a central processing unit (CPU) or image processor 1 for adapting a deformable 

10 model surrace to surfaces of multiple or composed objects by mesh adaptation. 

Preferably, this adaptation is performed simultaneously for a plurality of objects in an 
image. The image processor 1 is connected to a memory 2 for storing the deformable 
surface model comprising a first and a second partial deformable surface model, and for 
storing the image depicting the objects of interest. The image processor 1 may be 

1 5 connected via a bus system 3 to a plurality of periphery devices or input/output devices 
which are not depicted in Fig. 1. For example, the image processor 1 may be connected 
to a MR device, a CT device, to a plotter or a printer or the like via bus system 3. 
Furthermore, the image processor 1 is connected to a display such as a computer screen 
4 for outputting segmentation results or information. Furthermore, a keyboard 5 is 

20 provided, connected to the image processor 1 , by which a user or operator may interact 
with the image processor 1 or may input data necessary or desired for the segmentation 
process. 

Figure 2 shows a flow chart of a method or computer program according to an 
25 exemplary embodiment of the present invention which is executed on an image 
processing device shown in Figure 1. 

After initiaUsation in step 100, in step 101, a deformable model M and an image I 
containing a structure of interest are loaded from the memory 2 into the image processor 
30 1 . Then, the model is coarsely placed over the object of interest in the image I. This can 
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be done, e.g. by a user using the keyboard 5, a mouse or a similar interactive device, or a 
suitable automated initial placement method as known in the art. 

The composed deformable model M represents surfaces of several anatomical objects 
5 and their spatial constellation. Fig. 3 shows an example of the surface model of two 
vertebrae. Here, the deformable surface model M comprises the first partial deformable 
surface model to be adapted to the surface of the first vertebra and the second partial 
deformable surface model to be adapted to the surface of the second vertebra. In spite of 
the fact that two vertebrae are depicted, other structures, like e.g. the femur and the hip, 
10 or the inner or outer wall of the heart ventricle are possible. In case, more than two 
objects are to be segmented simultaneously, a number of partial deformable surface 
models may be provided. 

The depicted composed first and second partial deformable models consist of the 
15 surfaces of the two vertebrae. Each object is represented by a polygonal mesh. In the 
present example, vertices xj are connected by triangles tj. Nevertheless, other 
representations, such as simplex or general polygonal meshes are possible, Such a 
surface representation can be derived by a triangulation method of a single training 
object, as described, for example, in W.E. Lorensen, et al. "Marching cubes: A high 
resolution 3D surface construction algorithm" Computer Graphics, 21(3): 163-169, 
1987. 

In Figure 4, according to an exemplary embodiment of the present invention, a priori 
knowledge of the spatial relationship of the two vertebrae is included in the model As 
shown in Fig. 4, the model was augmented with additional edges which connect vertices 
of the first vertebra with vertices on the second vertebra. In Fig. 4, these additional 
dedicated connections between neighbouring triangles of the meshes of the two 
vertebrae are clearly visible between, e.g. the top endplate of the lower vertebra and the 
lower endplate of the top vertebra. These dedicated connections represent a priori 
knowledge of the spatial relationships between the objects. These connections between 
vertices of the neighbouring vertebrae are derived as follows: 
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Given two objects d and Oz (the two vertebrae) each with a set of vertices xu and xb . 
For each vertex xu of object d, the closest vertex x a of object O2 is searched. This 
results in a list of vertex pairs {U, x, kl ), (x n , x lk2 ), ...}. For each vertex pair, a length | 
xoi - xiki I is calculated and only those pairs with a length smaller than a certain threshold 
5 Tare kept. This results in the feet, as shown in Fig. 4, that connections are only 

introduced between vertices which are not too far distanced from each other. As can be 
taken from Figure 4, with the adjustment of the threshold T, connections are only 
introduced between neighbouring triangles of the two vertebrae. The resulting pairs are 
stored in the model as additional or "connection" triangles c fc where point 0 and point 2 
10 are the vertex xoi of object O,, and point 1 of the triangle is the corresponding vertex 
xiu of object Cb. This is depicted in further detail in Figure 5. 

In Figure 5, there is a first triangle 10 on the surface of object Oi and a second triangle 
20 on the surface of the object 0 2 . For the sake of clarity, only one triangle of the mesh 

1 5 is depicted for each object Oi and 0 2 . Ascan be taken from Fig. 5, the vertex Xoi on the 
surface of object Oi is connected with the vertex x m on the surface of object 0 2 . This 
connection is the additional or the "connection" triangle The triangle 20 shows a 
numbering of the corners of the triangle 20. The corner of the triangle 20 of the vertex 
Xlk i is designated with 0', the lower corner on the left side is designated with 1' and the 

20 lower corner on the right side of the triangle 20 is designated with 2'. Using this way of 
numbering the corners of the triangle, it can be taken from Figure 5 that the corners 0 
and 2 of the triangle are the vertex xoi and the corner 1 of the triangle c k is the vertex 
xiki. Due to this, the triangle has no surface. In other words, the connection triangle c k 
is a degenerated triangle, meaning that the edge between points 0 and 2 has a zero 

25 length, and the edge between points 0 and 1 is identical with the edge between points 2 
and 1. Also, the edges between 0 and 1, respectively, 2 and 1 are at the same distance as 
the vertices Xoi and x m -. Hence, in total, the model consists of a set of surface voxels 
{x;}, a set of surface triangles {tj}, and a set of connection triangles {c k }. Accordingly, 
as may be taken form Figure 4, in the deformable model, additional connection triangles 

30 between the upper and the lower vertebra were added to the surface triangles. 
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Having generated the composed deformable model and initialised the starting position in 
the image, an iterative adaptation procedure is carried out in steps 102 and 103. Here, 
the first and second partial deformable surface models are adapted to the first and second 
objects, i.e. the two vertebrae. The procedure iteratively varies the vertices coordinates 
5 so that they optimally fit the surface of the structure of interest, i.e. the two vertebrae. 

To adapt the deformable surface model to the object in the image, the method of T. 
Mclnterney et al. may be used. However, preferably a modified method based on Weese 
et al. is used. For a better understanding, the modified method by Weese et al. is briefly 
10 summarized for single objects, and then the extension to multiple objects with 
connection triangles is explained. 



Given an initial surface model of the object, each iteration consists of a feature detection 
step (step 102) and a mesh reconfiguration step (step 103). In step 102, for each of the 
15 triangles, a search along the triangle normal is carried out to find a point x that 
potentially lies on the boundary of the object in the image. 

In step 103, the position of the mesh vertices are recalculated by energy minimization 
with respect to the vertex coordinates. The energy is a weighted sum of two terms, 



20 



E = £ cxt + oE int (i) 



The external energy 2^* drives the mesh vertices towards the surface elements 
associated with the detected feature points and can be calculated as in Weese et al. 

The internal energy restricts the movement of the mesh vertices. Since the distribution 
of vertices on the initial template mesh is optimal with respect to surface-curvature, 
advantageously that distribution is maintained by punishing the change of distance 
between neighbouring vertices. Therefore, it is defined: 



30 
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(2) 

where N is the number of objects, ti is the set of vertices belonging to the i-th object, Vj 
is the number of vertices of the i-th object, x k and *j represent two neighboring 
5 vertices of the deformable mesh as found in step 102, and A jk is the corresponding 

vectorial difference between vertex j and its neighbour k of a reference model. A scaling 
s and a rotation matrix R between the initial model and the deformed model are also 
estimated. 

10 The result of this minimization are new vertex coordinates, which reflect a trade-off 

between being close to the surface points detected in the image and staying similar to the 
initial deformable model. From these new vertex coordinates new boundary points are 
searched for in step 102, and so forth. These two steps are iterated a fixed number of 
times, leading to a deformed model close to the surface of the object of interest. 

15 

In the following, the method of Weese et al. will be extended for the adaptation of 
composed deformable models. If the structure of interest is composed of several objects 
which are not connected but spatially related in a certain way, correct adaptation may 
become a problem. The spatial relationships may be violated during mesh 
20 reconfiguration, leading to meshes which are intersecting or too close to each other. 
One of the main reasons is that no information about the spatial coupling of the vertices 
of the different meshes is taken into account during adaptation in the known methods. 

According to an exemplary embodiment of the present invention, in order to maintain 
25 certain spatial relationships between the meshes, "connections triangles" are introduced, 
in order to approximately maintain the distance between the two objects, or referring to 
the examples of Figs. 3 and 4, to maintain the spatial relationships between the two 
vertebrae. As may be seen from the following, the spatial relationship is used during 
adaptation. 



/ j=\J&,keN t (f> 
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As described above, according to an exemplary embodiment of the present invention, 
additional connection triangles Ck, each of which connects a vertex of the lower vertebra 
to a vertex of the upper vertebra, are introduced. In order to approximately maintain the 
initial distances between the vertebrae, preferably, the initial configuration of the 
5 connection triangles is maintained. This can be done by adding an additional energy (see 
equation (1)), i.e. 

£ = Eext +0(£j n t + -^connections) 

3 The purpose of ^connections is to restrict the change of the connection triangles. The 
connection energy can be formulated as 



N V, 

Ejection, = 2 S ~ S ^Jkf> (4 . 

Here, only those vertices are considered which belong to a set of connection triangles 
1 5 Accordingly, only those vertex neighbours N 0 (j) are considered, which belong to the 
same connection triangle as the vertex Xj. A* is the vectorial distance between two 
points of a connection triangle. 

It is important that only triangles are introduced, but no additional vertices (note that this 
20 it true because the additional triangles were generated from the available surface 

vertices). The internal energy^ and the connection energy ^connection are combined into 
one internal energy 



N V t 

E iM = E,„, + EeonuecUon, = S 2 ^y^J~^k~ S ^jk) 2 } ( 5 \ 
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This leads to the new energy function of the composed deformable model 

E = E ext +aE lat ( 6 ) 

5 Then, the energy is minimized. Here it should be noted again that due to the fact that 
the connection triangles do not have a surface, no feature search is carried out in step 
102 for the connection triangles Thus, a feature search is only carried out for the 
surface triangles tj. This advantageously allows to minimize the computation efforts. 

10 Then, for displaying the deformable model, the vertices x 5 and the triangles tj can be 
selected in step 104 and displayed on the computer screen 4. However, it should be 
noted that instead of displaying the entire deformed model, an output of the vertex 
coordinate numerically is also possible. Then, the method continues to step 105, where 
the method ends. 

15 

In brief, according to the present invention, the spatial relationship of objects is used 
during adaptation of the deformable surface model to the objects. Accordingly, due to 
the inclusion of additional connections to the vertices into the shape model, which 
vertices only contribute to the internal energy, but are not used to detect feature points, 

20 and therefore do not contribute to the external energies, spatial relationships between 
multiple or composed objects can be maintained very easily by minimizing the internal 
energy. This allows to accurately maintain spatial relationships between objects which 
are simultaneously segmented. In particular, the danger is prevented that spatial 
relationships between objections are violated and that the composed deformable models 

25 intersect each other. A segmentation of multiple objects at the same time also improves 
the robustness and speed of segmentation using deformable models. 



30 
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1. Method for the simultaneous segmentation of multiple or composed objects in an 
image, wherein a deformable surface model is to be adapted to a first surface of a first 
object and a second surface of a second object and wherein the deformable surface 
model comprises a first partial deformable surface model and a second partial deformable 
surface model, comprising the steps of: 

(a) applying the first partial deformable surface model describing a structure of 
the first surface of the first object; 

(b) applying the second partial deformable surface model describing a structure 
of the second surface of the second object, wherein the first partial deformable surface 
model and the second partial deformable surface model have a prescribed spatial 
relationship corresponding to a spatial relationship of the first object and the second 
object; and 

(c) adapting the first partial deformable model to the first surface and the second 
partial deformable model to the second surface, wherein the prescribed spatial 
relationship of the first partial deformable surface model and the second partial 
deformable surface model is used for the adaptation. 

2. Method according to claim 1, wherein the spatial relationship of the first partial 
deformable surface model and the second partial deformable surface model is prescribed 
by means of an additional edge, which connects a first vertex of the first partial 
deformable surface model with a second vertex of the second partial deformable surface 
model. 
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3. Method according to claim 2, wherein the additional edge is a featureless vertex 
connection. 

4. Method according to claim 1, wherein the first and second partial deformable surface 
5 models each comprise a mesh with a plurality of surface elements, further comprising the 

steps of: 

detecting feature points for the surface elements at the first and second surfaces 

of the first and second objects; and 

recalculating coordinates of the surface elements of the mesh to represent the 

10 feature points. 

5. Method according to claim 4, wherein the recalculation step comprises the steps of: 

minimizing a distance between the feature points and the surface elements; and 
niimmizing an internal energy of the first and second partial deformable surface 

15 models. 

6. Method according to claim 5, wherein the internal energy comprises an extended 
internal energy relating to a difference of a length of the additional edge and a distance 
between the first and second partial deformable models. 

20 

7. Image processing device, comprising: 

a memory for storing a deformable surface model comprising a first deformable 
surface model and a second deformable surface model and for storing an image depicting 
a first object and a second object; and 
25 an image processor for adapting the deformable surface model to a first surface 

of the first object and a second surface of the second object, which processor performs 
the following operation: 
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(a) applying the first partial deformable surface model describing a structure of 
the first surface of the first object; 

(b) applying the second partial deformable surface model describing a structure 
of the second surface of the second object, wherein the first partial deformable surface 

5 model and the second partial deformable surface model have a prescribed spatial 
relationship corresponding to a spatial relationship of the first object and the second 
object; and 

(c) adapting the first partial deformable model to the first surface and the second 
partial deformable model to the second surface, wherein the prescribed spatial 

10 relationship of the first partial deformable surface model and the second partial 
deformable surface model is used for the adaptation. 

8. Computer program for an image processing device in accordance with claim 6, for 
adapting a deformable surface model comprising a first partial deformable surface model 
15 and a second partial deformable surface model to a first surface of a first object and a 
second surface of a second object, comprising the following steps: 

(a) applying the first partial deformable surface model describing a structure of 
the first surface of the first object; 

(b) applying the second partial deformable surface model describing a structure 
20 of the second surface of the second object, wherein the first partial deformable surface 

model and the second partial deformable surface model have a prescribed spatial 
relationship corresponding to a spatial relationship of the first object and the second 
object; and 

(c) adapting the first partial deformable model to the first surface and the second 
25 partial deformable model to the second surface, wherein the prescribed spatial 

relationship of the first partial deformable surface model and the second partial 
deformable surface model is used for the adaptation. 
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Simultaneous Segmentation of Multiple or Composed Objects by Mesh Adaptation 

Deformable models are used for the segmentation of structures in 3D images. The basic 
principle of such methods consists of the adaptation of flexible meshes to the image. 
5 However, the simultaneous segmentation of multiple or composed objects often causes 
problems in that spatial relationships between the objects are violated, or that meshes are 
intersecting each other. According to the present invention, a priori knowledge about 
spatial relationships between objects is introduced into the shaped model. This allows to 
maintain spatial relationships between the objects and to avoid intersecting meshes. 



10 



Fig. 2 
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